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Abstract 

Twelve air transport-rated pilots participated as 
subjects in a motion-base simulation experiment to 
evaluate the use of external Vision Systems (XVS) as 
enabling technologies for future supersonic aircraft 
without forward facing windows. Three head-up flight 
display concepts were evaluated -a monochromatic, 
collimated Head-up Display (HUD) and a color, non- 
collimated XVS display with a field-of-view (FOV) 
equal to and also, one significantly larger than the 
collimated HUD. Approach, landing, departure, and 
surface operations were conducted. Additionally, the 
apparent angle-of-attack (AOA) was varied (high/low) 
to investigate the vertical field-of-view display 
requirements and peripheral, side window visibility 
was experimentally varied. The data showed that 
lateral approach tracking performance and lateral 
landing position were excellent regardless of AOA, 
display FOV, display collimation or whether peripheral 
cues were present. However, the data showed glide 
slope approach tracking appears to be affected by 
display size ( i.e ., FOV) and collimation. The 
monochrome, collimated HUD and color, uncollimated 
XVS with Full FOV display had (statistically 
equivalent) glide path performance improvements over 
the XVS with HUD FOV display. Approach path 
performance results indicated that collimation may not 
be a requirement for an XVS display if the XVS 
display is large enough and employs color. Subjective 
assessments of mental workload and situation 
awareness also indicated that an uncollimated XVS 
display may be feasible. Motion cueing appears to 
have improved localizer tracking and touchdown sink 
rate across all displays. 

Introduction 

NASA is conducting research into technologies 
for reducing the impact of aircraft sonic boom on 
people and the environment. The primary objective of 
this research is to enable regulatory changes that would 
permit unrestricted supersonic flight overland, both 
domestically and internationally. A successful low- 


boom design drives the shaping and configuration of 
the vehicle. One such conceptual configuration is 
shown in Figure 1. As evident in this figure, the 
forward visibility for the pilot/flight crew is severely 
compromised as a result of the vehicle shaping. 

Under the Fundamental Aeronautics (FA) 
Program, Supersonics project, NASA is performing 
fundamental research, development, test and evaluation 
of flight deck and related technologies to support these 
low-boom, supersonic configurations by use of an 
external Vision System (XVS). XVS is a combination 
of sensor and display technologies intended to provide 
an equivalent level of safety and performance to that 
provided by forward-facing windows in today’s 
aircraft. 



Figure 1. Conceptual Low-Boom Supersonic 
Aircraft Configuration 

Without XVS, the economic viability of a low- 
boom supersonic aircraft is questionable, since the lack 
of forward visibility by the pilot would severely restrict 
aircraft operations and airspace usage especially when 
the weather is clear and visibility conditions are 
unrestricted - i.e., without an XVS, a low-boom 
supersonic aircraft cannot operate under Visual Flight 
Rules (VFR) since it can’t “see-and-avoid” and “see- 
to-follow.” 

“Sense-and-avoid” technologies, in lieu of see- 
and-avoid, are actively being pursued in the 
Uninhabited Air Vehicle (UAV) sector and their work 
is directly applicable [1], To date, however, these 
concepts are immature and will unlikely be advanced 
enough to support the operating concepts and airspace 
needs of a commercial business aircraft [2]. 



Acceptance of these technologies by owners and 
operators of commercial and business aircraft is also 
debatable. 

Significant research was conducted under 
NASA’s High Speed Research program during the 
1990s on the design and development issues associated 
with an XVS for a conceptual high-speed civil 
transport aircraft [3,4]. What emerged from this 
research (and still holds true today) is that the key 
challenge for an XVS design exists during VFR 
operations. The driving XVS design standards emerged 
from the three tenets of VFR operations which are 
“see-and-avoid,” “see-to-follow,” and “self- 
navigation.” These VFR-type requirements are not 
unique to low-boom supersonic aircraft but the absence 
of natural forward vision creates the equivalent 
performance and safety requirements for an XVS 
design [2], 

An experiment was conducted to evaluate some of 
the design requirements of an XVS - without the need 
for forward-facing windows - and to determine the 
interaction of XVS and peripheral vision cues for 
terminal area and surface operations. Another key 
element of the testing investigated the pilot's awareness 
and reaction to non-normal events (i.e., failure 
conditions) that were unexpectedly introduced into the 
experiment. These non-normals are critical 
determinants in the underlying safety of all-weather 
operations. 

This paper describes an experimental evaluation 
of field-of-view (FOV), collimation, and peripheral 
cues on pilot performance and subjective ratings of 
situation awareness and workload during terminal area 
operations. In addition, motion effects are analyzed by 
comparing objective results from this motion-base 
simulation experiment to a previously conducted 
experiment [8] that manipulated the same independent 
variables using the same simulator in fixed-base mode. 
The subject pilots were different although the 
recruiting criteria were the same. Further, the 
objective data from this test are being used to develop 
performance-based approach and landing standards 
which might establish a basis for future all-weather 
landing certification. 


Method 

Subjects 

Twelve pilots, representing 8 airlines participated 
in the experiment. All participants had previous 
experience flying Head -Up Displays (HUDs). The 
subjects had an average of 1173 hours of HUD flying 
experience and an average of 16.8 years and 13.3 years 
of commercial and military flying experience, 
respectively. 

Simulation Facility 

The experiment was conducted in the Integration 
Flight Deck (IFD) simulator (Figure 2) on the motion- 
base platform using the Cockpit Motion Facility 
(CMF) at NASA Langley Research Center. The IFD 
emulates a Boeing B-757-200 aircraft and provides 
researchers with a full-mission simulator capability. 
The cab is populated with flight instrumentation and 
pilot controls, including the overhead subsystem 
panels, to replicate the B-757 aircraft. The collimated 
out-the -window (OTW) scene is produced by an Evans 
and Sutherland Image Generator 4530 graphics system 
providing approximately 200 degrees horizontal by 40 
degrees vertical FOV at 26 pixels per degree. The 
forward windows were masked for this experiment but 
the side windows were unblocked to test the effects of 
peripheral cues (with and without) during approach, 
landing, taxi, and departure operations. 

The evaluation pilot (EP) occupied the left seat, as 
the Pilot Flying (PF) for this experiment. The left seat 
included an overhead HUD projection unit and a 22 
inch diagonal liquid crystal display (LCD) referred to 
as the XVS display (Figure 2). The right seat was 
occupied by a principal investigator (PI) who acted as 
First Officer during data collection. The PI aided the 
EP by providing callouts during taxi and performing 
airplane configuration tasks during departure runs. 

Head-Up Display 

The HUD was collimated and subtended 
approximately 26° horizontal by 21° vertical FOV. 
Note that to maintain conformality with the outside 
world, the FOV for the HUD imagery was fixed and 
could not be varied by the EP. The HUD presentation 
was written strictly in raster format from a video 
source (RS-343) input. The input consisted of a video 
mix of symbology and a simulated camera image (i.e., 
XVS display). The symbology included “haloing” to 
ensure that the symbology was highlighted against the 



scene imagery background. Overall HUD brightness 
and contrast controls were provided to the pilot. In 
addition, the EP was able to independently adjust the 
flight symbology brightness relative to the raster 
imagery. The pilot also had a declutter control, 
implemented as a four-button castle switch on the left 


hand horn of the PF yoke. Four “declutter” states were 
available to the EP: 1) Symbology toggle (on/off); 2) 
Imagery toggle (on/off); 3) All decluttered (no 
symbology or imagery); and 4) All displayed (both 
symbology and imagery). 



Figure 2. IFD Simulator with HUD, XVS Display (interior view) and CMF with IFD (exterior view) 


external Vision System Display 

The XVS display subtended approximately 44° 
horizontal by 34° vertical FOV and was located 
approximately 19 inches from the pilot design eye 
point. The imagery on the XVS display was conformal 
with the OTW view just as the HUD imagery. 
However, this display differed from the HUD as it was 
larger, used color, and was not collimated. The XVS 
display emulated a camera view mounted on the 
outside of the aircraft with flight symbology overlaid 
on the scene. Thus, any items (e.g., traffic, approach 
lighting system, terrain, runway markings, etc.) that 
would be visible to a real camera system would be 
visible in the color camera imagery. This photo- 
realistic camera imagery was unaffected by the outside 
weather (similar to HUD SV imagery) to 
parametrically test for any interactions between display 
size and peripheral cues. The same declutter control 
described in the HUD section above was utilized with 
the XVS display. 

Symbology 

The same symbology set was used for the XVS 
and HUD concepts (Figure 3). The symbology 


included pathway guidance and a runway outline. The 
pathway symbology [9] ended at 500 ft HAT and was 
replaced by a runway outline and a glideslope 
reference line. A runway outline symbol (8000 ft x 200 
ft) was drawn using the threshold coordinates of the 
landing runway and the aircraft navigation solution to 
conformally position the symbol. A glideslope 
reference line was drawn at a descent angle of 3.1 
degrees. Also, radar altitude was shown digitally 
underneath the flight path marker when below 500 ft 
above ground level (AGL). 

A pitch-roll guidance cue (“ball”) used modified 
pursuit guidance along the desired path [10]. 
Horizontal and vertical position of the ball reflects the 
track and vertical flight path angles to fly to the center 
of the desired path. The path deviation indicators 
showed angular course deviation (i.e., glideslope and 
localizer-like) conditions by converting the linear path 
error data to angle errors and scaling in “dots.” 
Glideslope and localizer raw data indicators which 
included a deviation scale and angular deviation 
indication were also provided (i.e., glideslope and 
localizer deviation). 



Figure 3. Head-Up Flight Display Symbology 
Format - Low AO A Condition Shown 


Independent Variable - Display Concepts 

Four head-up flight display concepts were 
evaluated by the EPs while flying approaches to 
Runway 16R at the Reno-Tahoe International Airport 
(airport identifier RNO). The head-down PFD and ND 
formats were invariant. 

Head-Up Flight Display Concepts 

The four head-up flight display concepts (Figure 4 
and Table 1) were a partial factorial combination of 
angle-of-attack, or AOA (low or high) and display type 
(HUD, XVS display with HUD FOV, or XVS display 
with Full FOV). Specifically, all three display types 
were flown in the low AOA condition and only the 
XVS display with Full FOV was flown in the high 
AOA condition. 

The HUD installed in the RFD simulator uses a 
pitch bias value of 3 degrees. This means that the 
waterline (boresight) reference point of the HUD is 3 
degrees above the center of the HUD. This bias 
optimally tailors the placement of the HUD symbology 
for the B-757 simulator in its nominal operating flight 
conditions. In Figure 3, the HUD during this “low 
AOA” condition is shown. With the B757 at 
approximately 3 degrees in the approach conditions for 
this test, the flight path marker on short final is 
approximately at the center of the HUD. 

One of the display design issues facing a 
supersonic aircraft is that they typically operate at high 
AOA conditions in the approach and landing, due to 
the high sweep-back wing angles and low camber 


typically desired for efficient supersonic cruise flight. 
For the flight deck designer, high AOA conditions on 
the approach drive the vertical FOV of windows to 
allow pilot visibility of the approach lights and 
touchdown zone (see FAA Advisory Circular AC25- 
773-1 - Reference 11). For the Concorde, a drooped- 
nose was used to provide this visibility. By analogy, 
an XVS display, providing equivalent visibility, would 
require the same vertical FOV, particularly if 
conformal symbology and imagery create “electronic” 
visibility. As the AOA becomes very large, the 
recommended down-angle becomes significantly 
larger, following the so-called “3-second rule.” 
Review of AC25-773-1 suggests that the substantiation 
for this requirement is vague and dated, particularly its 
relevance, as it might be applied to “electronic” 
visibility systems [2], 

As an initial evaluation of this FOV down-angle 
requirement, a “high AOA” condition was simulated to 
compare to the “low AOA” condition. To minimize a 
confound in the experiment, actual high angle-of-attack 
conditions for the B757 were not simulated; otherwise, 
the flying qualities of the low AOA and high AOA 
conditions would be radically different. Instead, a pitch 
bias was introduced. 

The high AOA condition used an 8° pitch bias to 
simulate an increased angle-of-attack to approximate 
that of a supersonic transport aircraft on approach 
(Figure 5), approximately 11° angle-of-attack. 
Comparing Figure 3 and Figure 5 shows the 
differences in head-up flight display symbology. The 
pitch bias caused a “symbology cluster” with the flight 
path marker, guidance cue and localizer deviation 
scale/marker while the pilots performed an approach 
on a 3. 1 degree glide path to Runway 16R. 

The two AOA conditions were evaluated 
independently on the uncollimated XVS display with 
the full FOV (44 x 34 degrees) condition. The 
symbology used in the XVS concepts was identical to 
that used in the HUD concept. 

The XVS and HUD concepts were located in the 
same head-up positions so the aircraft boresight 
references for each display were co-located. 





Figure 4. Head-Up Flight Display Concepts 


Simulated color camera imagery was mixed with 
the symbology and shown conformally on the color, 
uncollimated XVS display for both FOVs. The HUD 
was stowed to preclude blocking or distortion of the 
pilot’s forward view when using the XVS display. The 
XVS display was turned off when the EP was 
evaluating a HUD concept. Note that the forward 
windows were masked for both display devices and the 
side windows were unobstructed. 

Head-Down Flight Display Concepts 

The PFD and ND closely resembled current 
transport aircraft equipage. However, guidance 
information was purposely removed from the PFD so 
that the EPs would focus on the head-up primary flight 
display concepts. The ND showed the RNO Runway 
16R approach path, but it did not include any 
Enhanced Ground Proximity Warning System or 
Traffic Alert and Collision Avoidance System 
information. 
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Figure 5. Head-Up Flight Display Symbology 
Format - High AO A Condition Shown 


Independent Variable - Peripheral Cues 

To test for peripheral cue effects during 
approach/landing, surface and departure operations, 
two visibility levels were tested. The peripheral cues 
were either absent - simulating IMC of 200 ft runway 
visual range (RVR) - or present - simulating Visual 
Meteorological Conditions (VMC) of 3 miles visibility. 


Independent Variable - Motion Cues 

To test for motion cue effects during 
approach/landing, surface and departure operations, 



objective measures from this motion-base experiment 
were compared to a previously conducted XVS fixed- 
base IFD simulation experiment that manipulated the 
same independent variables [8] . 

Evaluation Tasks 
Approach 

The approach task mimicked an existing visual 
arrival procedure reflecting an efficient and preferred 
routing for air traffic control and noise abatement. 
This approach normally requires visual flight 
conditions for the crew to see-and-avoid terrain, traffic, 
and obstacles while navigating with respect to ground 
references. The approach was a curved, descending 
path around terrain and obstacles and, thus, tests the 
ability of the display concepts to support this type of 
equivalent visual operation. The weather consisted of 
altitude -based cross winds (wind direction and 
intensity was dependent on altitude), light turbulence, 
and varying visibility levels (3 miles or 200 ft RVR). 
The EP hand-flew the base and final legs of the visual 
arrival to RNO Runway 16R, using the HUD or XVS 
display concept with autothrottles engaged, holding 
132 knots. The aircraft was configured for landing 
(landing gear down and flaps 30 degrees) prior to each 
run, and the aircraft was “cleared to land”. The path 
converged into the instrument landing system approach 
course, nominally resulting in a stabilized approach no 
lower than 1,000 ft HAT. For the low AOA runs, the 
pilot was instructed to follow a pre -briefed taxi 
clearance requiring the aircraft to exit the runway on a 
high-speed turnoff onto Taxiway November, turn right 
on Taxiway Alpha, cross over Runway 7/25, and then 
turn left on Taxiway Lima where the run ended. For 
the high AOA runs, the pilot was instructed to come to 
a full-stop on Runway 1 6R where the run ended. 

Departure 

EPs also performed departures flying the RNO 
“Mustang 7” Departure Procedure. They maintained 
the runway heading of 168 degrees until waypoint 
RIJTU (about 5 nmi from the departure runway) and 
then turned left direct toward the Mustang VORTAC, 
where the run ended. The weather consisted of altitude- 
based cross winds (wind direction and intensity was 
dependent on altitude), light turbulence, and varying 
visibility levels (3 miles or 200 ft RVR). The EP hand- 
flew the departure with the HUD, XVS with HUD 
FOV or XVS with Full FOV display concept and was 
instructed to climb to 10,000 ft mean sea level (MSL) 
and 250 knots. 


There were up to 3 transport-sized aircraft in the 
runway environment, but they did not provide any 
conflicts for the ownship during approach, landing, 
taxi, or departure operations. There was no Air Traffic 
Control involvement in the tasks. 

Pilot Procedures 

Since only pilot-flying evaluations were being 
conducted, automatic aural altitude call-outs (e.g., 
1000, 500, 100 feet, etc.) were included in the 
simulation to “assist” in altitude awareness. Unlike 
current FAA regulations, for this experiment, the EP 
was not required to see using natural vision the 
required landing visual references (as per FAR 
§91.175) by DA/H (decision altitude/height). The EP 
was instructed to continue to landing if the required 
landing visual references were seen in the imagery on 
the HUD or XVS and if the EP determined that a safe 
landing could be performed. Otherwise, a missed 
approach should be executed. 

The EPs were instructed to fly the aircraft as if 
there were passengers aboard, fly the center of the 
approach path (within + Vi dot for desired performance 
and within ± 1 dot for adequate performance), and land 
as close as possible to the centerline and aim point 
(1000 feet from the threshold). After landing, they 
were to capture the center line and then for the low 
AOA runs only, taxi at a speed with which they were 
comfortable using the pre-briefed taxi clearance. They 
were also instructed to initiate a go-around if the 
landing was not safe or if there were any safety 
concerns during the approach. EPs were instructed to 
stop the aircraft if they felt unsafe during surface 
operations. 

Prior to run commencement, the EP was briefed 
on the type of run to be completed, the display concept 
to be evaluated, the visibility level, and the wind 
magnitude and direction. 

Experiment Matrix 

Nominally, ten training runs and twenty-three 
experimental runs were completed by each EP. Of the 
23 experimental runs, 5 non-normal runs were included 
to investigate the pilot's awareness and reaction to 
unexpected events and conditions (e.g., failures). The 
non-normal data are critical determinants in the 
underlying safety of all-weather operations. Due to 
paper page-limit constraints, these data are not reported 
herein. 



For approach and landing runs, the experiment 
matrix (Table 1) consisted of a partial-factorial 
combination of display type (HUD, XVS with HUD 
FOV, or XVS with Full FOV), AOA (low or high) and 
peripheral cues in side windows (absent or present). 
The MALSR ALS (Medium-Intensity Approach 
Lighting System with Runway Alignment Indicator) 
was held fixed for the approach and landing runs listed 
in Table 1. 


Table 1. Approach and Departure Run Matrix 



HUD 

XVS with 
HUD FOV 

XVS with 
Full FOV 

VMC Low AOA 
Approach 

S 


V 

IMC Low AOA 
Approach 



V 

VMC High AOA 
Approach 


S 


IMC High AOA 
Approach 


S 


VMC Departure 

V 

S 

V 

IMC Departure 

V 


V 


Four additional runs were conducted to test for 
display type (HUD or XVS with Full FOV) and 
approach lighting system (VFR, ALSF-2 [Approach 
Lighting System with Sequenced Flashing Lights]) 
effects. The low AOA condition and the IMC visibility 
level were held fixed for these comparisons. These 
four runs were compared to the 2 analogous MALSR 
ALS runs (HUD and XVS with Full FOV) from the 
Table 1 experimental matrix to test for ALS effects. 
Due to page limitations, the results for the ALS 
analyses are not reported herein. 

For departure runs, the experiment matrix (Table 
1) consisted of a full -factorial combination of display 
type and peripheral cues in the side windows for a total 
of 6 runs. Only low AOA conditions were flown for 
the departure runs. 

Measures 

During each approach and landing run, path error, 
pilot control inputs, and touchdown performance (fore 
or aft of touchdown zone, and distance left or right of 
centerline) were analyzed. During taxi operations, 
centerline tracking and taxi speed were measured. For 


departure runs, centerline tracking, heading and climb 
rate maintenance, and altitude capture were measured. 

After each run, pilots completed a run 
questionnaire consisting of the NASA Task Load Index 
(TLX) workload rating [12], Situation Awareness 
Rating Technique (SART)[12], and six Likert-type (5- 
point) questions specific to different constructs of 
making a stabilized and safe approach to landing, 
taxiing (when appropriate), or departure. 

After data collection was completed, pilots were 
administered two paired comparison tests: the Situation 
Awareness - Subjective Workload Dominance (SA- 
SWORD) [13] and Subjective Workload Dominance 
(SWORD) [12] techniques. The pilots also completed 
a post-test questionnaire to elicit comments on using 
the different display concepts, with and without 
peripheral cues, for conducting 1) low and high AOA 
approaches without a visual segment, 2) surface 
operations and 3) departures. 

Test Conduct 

The subjects were given a 1-hour briefing to 
explain the experiment purpose, HUD and XVS 
concepts, pilot procedures, and the evaluation tasks. 
After the briefing, a 1 -hour training session in the IFD 
was conducted to familiarize the subjects with the 
aircraft handling qualities, display symbologies, pilot 
procedures, and controls. The pilot’s responsibility for 
maintaining safe operations at all times was stressed. 
Data collection lasted approximately 4.5 hours and was 
followed by debriefings which included a final 
questionnaire. The entire session including lunch and 
breaks lasted approximately 8 hours. 

Results 

For the approach path error (deviation) data and 
landing performance data, repeated-measures analyses 
of variance (ANOVAs) were conducted for the factors 
of display type (HUD, XVS with HUD FOV, XVS 
with Full FOV) and peripheral cues (absent, present) 
for the fixed low AOA and MALSR ALS condition. 
Repeated-measures ANOVAs were also conducted on 
the XVS with Full FOV path deviation data to assess if 
there were AOA or peripheral cue differences for 
localizer and glide slope tracking. Additionally, 
motion (with, without) effects on path deviation data 
and landing performance data were evaluated by using 
a 2x3x2 mixed-factorial ANOVA with motion as the 
between-subjects factor and display type and 




peripheral cues as the within-subjects factors. The 
‘without’ motion runs were collected in a previously 
conducted fixed-base IFD simulation experiment [8] 
that manipulated the same independent variables as the 
motion-base IFD experiment reported herein. 

For the post-test paired comparisons, motion 
(with, without) effects on SA and workload were 
evaluated by using a 2x4 mixed-factorial ANOVA with 
motion as the between-subjects factor and display type 
as the within-subjects factor. 

Approach Performance 

Approach performance was assessed using rms 
(root mean square) localizer deviation (in dots) and 
rms glide slope deviation (in dots). These parameters 
correspond intuitively to the establishment and 
maintenance of a stabilized approach to landing - an 
important safety measure. 

The approach data were analyzed from 1000 ft to 
100 ft FIAT for the normal runs that ended in a landing. 
The beginning altitude value was chosen since the 
pilots were instructed to have the aircraft stabilized on 
the approach by 1000 ft FIAT else, they should perform 
a go-around. 

Both the rms localizer and rms glide slope 
deviation data had non-normal distributions. 
Logarithmic -transformed rms localizer and glide slope 
deviation data provided normal distributions and were 
used in the repeated-measures ANOVAs for the path 
deviation data. Mauchly’s test showed that the 
condition of sphericity (equal differences between data 
taken from the same participant) had been met for each 
of the ANOVAs conducted on the path deviation data. 

When means of the logarithmic path deviation 
data are transformed back to original units, the mean 
reported is actually the geometric mean [14] of the path 
deviation data and that is what is reported herein. 

Localizer Angular Deviation 

Display Type/Peripheral Cues Effects 

The interaction between display type and 
peripheral cues was statistically significant (F(2, 
22)=7.513, p<.01) for rms localizer deviation (Table 
2). Pilots flew a more precise lateral path when 
peripheral cues were absent (IMC condition) compared 
to when there were present (VMC condition) with the 
HUD and XVS with Full FOV displays; however, the 
opposite trend was true for the XVS with HUD FOV 


display where pilots flew more precise laterally when 
peripheral cues were present compared to when they 
were absent. 


Table 2. Approach Path Deviation 



HUD 

XVS w/ 
HUD FOV 

XVS w/ 

Full FOV 

IMC 

VMC 

IMC 

VMC 

IMC 

VMC 

Geometric Mean 

rms 
loc dev 
(dots) 

0.01 

0.02 

0.02 

0.01 

0.02 

0.02 

rms 
gs dev 
(dots) 

0.14 

0.15 

0.25 

0.20 

0.17 

0.12 


AOA/Peripheral Cue Effects 

There were no significant (p>0.05) differences 
between the main factors, AOA and peripheral cues, or 
their second-order interaction for rms localizer 
deviation (overall geometric mean=0.018 dots). 

Motion Effects 

Motion was significant for rms localizer 
deviation, F(l,33)=9.820, p<0.01. Pilots had 

significantly better lateral path tracking when they flew 
with motion (geometric mean=0.017 dots) compared to 
when they flew without it (geometric mean=0.027). 
There were no significant (p>0.05) second-order 
motion effects for this measure. 

Glide Slope Angular Deviation 

Display Type/Peripheral Cues Effects 

Display type (F(2,22)=15.588, p<0.001) was 
significant for rms glide slope deviation, but peripheral 
cues and the interaction between display type and 
peripheral cues was not significant (p>0.05). Pilots 
had significantly better glide path performance with the 
HUD and XVS with Full FOV display than with the 
XVS with HUD FOV display (Table 2). 

AOA/Peripheral Cue Effects 

Peripheral cues (F(l,l 1)=5.683, p<0.05) were 
significant for rms glide slope deviation but AOA and 
the interaction between peripheral cues and AOA were 
not significant (p>0.05) for this measure. Pilots had 
significantly better glide path performance when 




peripheral cues were present (geometric mean=0. 142 
dots) compared to when they were absent (geometric 
mean=0.177 dots). 



Figure 6. Logarithmic-transformed RMS Glide 
Slope Deviation Means for Display Type 

Motion Effects 

The second-order interaction between display and 
motion was significant (F(2,66)=5.85, p<0.01) for rms 
glide slope deviation (Figure 6). (Note that in Figure 
6, the logarithmic-transformed means are shown. 
When transformed back to original units, a more 
negative-valued logarithmic mean will indicate a 
lower-valued geometric mean than a less negative- 
valued logarithmic mean). Glide path tracking 
performance of the 3 display types differed in motion 
and fixed-base operations. Comparing motion to 
fixed-base runs, the HUD had improvements in glide 
path tracking and the two XVS displays had 
degradations in this measure. However, if you look at 
the motion run (logarithmic-transformed) means of rms 
glide slope deviation in Figure 6, nearly equivalent 
values are shown between the HUD and XVS with Full 
FOV displays. The main factor motion and the 
second-order interaction between peripheral cues and 
motion were not significant (p>0.05) for rms glide 
slope deviation. 


Objective Approach Standards Analysis 

The Joint Aviation Authorities (JAA) Joint 
Aviation Requirement (JAR) All Weather Operations 
(AWO) performance-based approach standard for go- 
around rate (AWO-202) in low-visibility approaches 
with decision heights below 200 ft and down to 100 ft 
was also applied in the objective data analysis [15]. 
Specifically, the standards specify that no more than 
5% of the approaches will have localizer deviations 
greater than 1/3 dot or glideslope deviations greater 
than 1 dot between 300 ft and 100 ft HAT for 
certification acceptance. These low-visibility approach 
standards were not written specifically as quantitative 
performance standards for advanced vision systems 
(such as XVS) operations, but are applied herein for 
comparative purposes. 

The Continuous Method [15] technique was 
employed to calculate the probability of success, P(a), 
of meeting the AWO exceedance criteria (1/3 dot 
localizer, 1 dot glideslope) with required levels of 
confidence with the different display concepts flown. 
The probabilities of success for meeting the AWO 
localizer and glideslope criteria are shown, broken 
down by AOA, display type and visibility condition, in 
Table 3 for this motion-base experiment. Also 
included in Table 3 are the applicable probabilities of 
success for a fixed-base IFD XVS experiment [8] that 
manipulated these same independent variables. 

The data in Table 3 shows that localizer tracking 
was maintained, irrespective of the display being 
flown, approach angle-of-attack, the absence or 
presence of peripheral cues in the side windows, or the 
absence or presence of simulator motion. For both the 
fixed-base and motion-base runs, the only display 
concept successfully meeting the JAR AWO-202 
localizer and glide slope criterion (greater than 95% of 
the time) was the low AOA XVS with Full FOV 
display concept flown with peripheral cues in the side 
windows (in bold, Table 3). This color, uncollimated, 
display concept showed conformal imagery over a 34° 
vertical FOV. 

There appears to be a motion/collimation effect in 
glide slope tracking performance. The collimated 
HUD concepts had improvements in glideslope 
tracking performance when the simulator was in 
motion compared to when it was in fixed-base; while, 
the uncollimated XVS with HUD FOV concepts had 
degradations in this measure when the simulator was in 
motion compared to when it was in fixed-base. There 



also appears to be a peripheral cue/uncollimated 
display effect for glide slope tracking. For the 
uncollimated XVS concepts, there were glide slope 
tracking improvements when peripheral cues were 
present compared to when there were none. There 


does not appear to be a motion effect on whether a 
pilot completed a landing or not as there were no go- 
arounds for the motion-base runs and only 1 go-around 
for the fixed-base runs. 


Table 3. Probabilities of Success in Meeting the AWO Localizer and Glideslope Criteria without a Visual 
Landing Segment in Fixed-Base and Motion-Base Simulation Runs 


Display 

AOA 

Wx 

Localizer P(a) 

Glideslope P(a) 

#Go-Around/#Total Runs 




Fixed- 

base 

Motion- 

base 

Fixed- 

base 

Motion- 

base 

Fixed- 

base 

Motion- 

base 

HUD 

Low 

IMC 

100 

100 

67 

88 

0/24 

0/12 

HUD 

Low 

VMC 

100 

100 

78 

83 

0/24 

0/12 

XVS HUD FOV 

Low 

IMC 

100 

100 

81 

55 

1/24 

0/12 

XVS HUD FOV 

Low 

VMC 

100 

100 

94 

72 

0/24 

0/12 

XVS Full FOV 

Low 

IMC 

100 

100 

93 

75 

0/24 

0/12 

XVS Full FOV 

Low 

VMC 

100 

100 

96 

96 

0/24 

0/12 

XVS Full FOV 

Hiqh 

IMC 

100 

100 

84 

74 

0/24 

0/12 

XVS Full FOV 

Hiqh 

VMC 

100 

100 

88 

81 

0/24 

0/12 


Note: Approaches using MALSR ALS. 

Landing Performance 

Landing performance was assessed using 
touchdown longitudinal position (in ft), lateral position 
(in ft) and sink rate (in ft/sec, or fps). All three 
touchdown measures had normal distributions. 
Mauchly’s test showed that the condition of sphericity 
had been met for each of the ANOVAs conducted on 
the landing data except for the motion effects second- 
order interaction of display and peripheral cues for 
touchdown lateral position, x 2 (2)=8.592. Therefore, 
degrees of freedom were corrected using Greenhouse- 
Geisser estimates of sphericity (s=.81). 

Display Type/Peripheral Cues Effects 

Display type (F(2,22)=7.065, p<0.01) was 
significant for touchdown longitudinal position. The 
pilots landed significantly further beyond the threshold 
when flying with the F1UD (mean=1823 ft) than when 
flying with the XVS with HUD FOV (mean=1496 ft). 
Neither the HUD nor XVS with HUD FOV were 
significantly different than the XVS with Full FOV 
(mean=1660 ft) for this measure. 

The interaction between display type and 
peripheral cues (F(2,22)=13.831, p<0.01) was 

significant for touchdown lateral position (Figure 7). 
Operationally though these differences were not 


significant as the largest lateral deviation from 
centerline was 6 ft. 

All other main effects and second order 
interactions were not significant (p>0.05) for the 
touchdown performance measures. 


Touchdown Lateral Position 



Figure 7. Lateral Touchdown Position - Display by 
Peripheral Cues 

Motion Effects 

Motion was significant for touchdown sink rate 
F( 1 ,32)— 34.52 1 , p<0.01. Pilots had significantly less 
sink rate when they flew with motion (mean=-3. 1 fps ) 
compared to when they flew without it (mean=-5.7 



fps). There were no significant (p>0.05) second-order 
motion effects for this measure. 

No motion effects were found for the touchdown 
longitudinal or lateral position measures. 

Objective Landing Standards Analysis 

Existing JAR AWO [15] performance-based 
landing standards (AWO 131) for longitudinal position 
and lateral position from centerline were applied in the 
objective landing data analysis. Specifically, the 
standards state that no longitudinal touchdown earlier 
than a point on the runway 200 ft from the threshold or 
beyond 2700 ft from the threshold and no lateral 
touchdown with the outboard landing gear more than 
70 ft from the runway centerline to a probability of 1 x 
10 . These standards pertain to the general concept of 
low-visibility approach and landings, but were not 
written specifically for operations with advanced 
vision systems such as XVS. 

This experiment used an aim point located 1000 ft 
from the runway threshold. For the simulated 757 
aircraft, the outboard landing gear would be 70 ft from 
the centerline when the fuselage (the recorded lateral 
landing position reported herein) is at 58 ft lateral 
deviation from centerline, assuming no crab angle at 
touchdown. In Figure 8, the touchdown data are 
shown, broken out by display concept and AOA 
condition but collapsed across peripheral cue influence. 
Included on this plot (in red, dashed rectangle) is the 
+/-58 ft lateral and 200 ft to 2700 ft longitudinal 
touchdown footprint defined in the JAR AWO landing 
criteria. 

Visual inspection of the data in Figure 8 show that 
all display concepts tested in motion, regardless of the 
AOA or peripheral cue condition, were within the JAR 
lateral and longitudinal touchdown criteria footprint. 
In contrast, the fixed-base study looking at the same 
display type/angle of attack concepts did not all meet 
the JAR touchdown criteria [8]. Specifically in that 
study, 4 HUD low AOA fixed-base runs landed beyond 
2700 feet from the threshold. 

These data were analyzed against the 1x10 6 
probability requirements. The analyses show that the 
lateral landing position met the 10 6 probability criteria 
(i.e., within 58 ft of centerline) for all the display 
type/angle of attack concepts presented in Figure 8. 


Satisfying the JAR AWO (low-visibility landing) 
longitudinal touchdown criteria to a 1x10 6 probability 
was not met with any of the display concepts tested. 

Approach and Landing Performance Discussion 

Elimination of the visual segment of the approach 
had no adverse affects on localizer tracking as it was 
excellent from 1000 ft to 100 ft HAT and well within 
JAR AWO approach criteria for decision heights below 
200 ft and down to 100 ft regardless of display size 
(large FOV or HUD FOV), AOA (low or high), 
collimation (with or without) or whether the pilot had 
peripheral cues or not. However, glide slope tracking 
from the required stabilized approach altitude of 1000 
ft to 100 ft HAT appears to be affected by display size 
(i.e., FOV) and collimation. The monochrome, 
collimated HUD and color, uncollimated XVS with 
Full FOV display had (statistically equivalent) glide 
path performance improvements over the XVS with 
HUD FOV display. This finding may indicate that 
collimation is not a requirement for an XVS display to 
have adequate approach path maintenance if the XVS 
display is large enough and employs color. In fact, JAR 
AWO glideslope tracking criteria was only met by the 
low AOA condition in the XVS with Full FOV display 
concept with peripheral cues in the side windows. 

Motion appears to improve glide path tracking for 
the collimated displays, lateral path tracking and 
touchdown sink rate for all the displays tested, and not 
degrade landing performance. All touchdowns 
occurred within the AWO touchdown box irrespective 
of display type or AOA when the simulator was in 
motion. 

The lateral touchdown data shows that the JAR 
AWO criteria were met with 1x10 6 probability for all 
motion concepts using the MAFSR AFS. Fateral and 
longitudinal positioning for touchdown was not a 
problem across any of the experimental display 
concepts. 

A widely perceived assumption is that peripheral 
cues are necessary for good flare and touchdown 
performance. This assumption was not supported by 
these data. No effect from the absence and presence of 
peripheral cues was found. This result supports other 
data [16] that peripheral cues are not as important on 
landing performance as generally assumed but a 
significant FOV in the forward field of view - as 
indicated by the large FOV XVS performance - is more 
influential. Further, these data highlight the 



importance of having motion cueing effects for 
representative sink rate at touchdown performance, 
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comparable to real-world performance expectations 
[16]. 
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Figure 8. Touchdown Data for Display Concepts for fixed MASLR ALS 


Mental Workload-SWORD 

Post-test, pilots were administered the paired- 
comparison SWORD scale that enabled comparative 
ratings of mental workload. Mental workload was 
defined for the pilots as “the amount of cognitive 
resources available to perform a task and the difficulty 
of that task.” The pair-comparison test was structured 
to compare the effects of color/collimation and 
peripheral cues (i.e., uncollimated, color XVS with 
HUD FOV flown in VMC; uncollimated, color XVS 
with HUD FOV flown in IMC; collimated, 
monochrome HUD flown in VMC; collimated, 
monochrome HUD flown in IMC) flown by the EP. 
Note that this comparison only considered the display 
types with the 26 deg x 21 deg FOV. 


Mauchly’s test indicated that the assumption of 
sphericity had been violated for the main factor display 
type, x 2 (5)=35.377. Therefore, degrees of freedom 
were corrected using Greenhouse-Geisser estimates of 
sphericity (e=.67). The post-test paired-comparison 
SWORD data indicated that display type 
(F(2. 00,67.98)^29. 610, p<0.01) was highly significant 
for the pilot ratings of mental workload. Contrasts 
showed three overlapping subsets for the mental 
workload ratings 1) uncollimated XVS with peripheral 
cues, uncollimated XVS without peripheral cues 
(lowest workload), 2) uncollimated XVS without 
peripheral cues and collimated HUD with peripheral 
cues, and 3) collimated HUD without peripheral cues 
(highest workload). The main factor motion and the 
second-order interaction between motion and display 
were not significant (p>0.05) for mental workload. 


Situation Awareness - SA-SWORD 

Post-test, pilots were administered the paired- 
comparison SA-SWORD scale that enabled 
comparative ratings of situation awareness. For these 
comparisons, SA was defined as “the pilot’s awareness 
and understanding of all factors that will contribute to 
the safe flying of their aircraft under normal and non- 
normal conditions.” The pair-comparison test was 
structured to compare the effects of collimation and 
peripheral cues (i.e,, uncollimated, color XVS with 
HUD FOV flown in VMC; uncollimated, color XVS 
with HUD FOV flown in 1MC; collimated, 
monochrome HUD flown in VMC; collimated, 
monochrome HUD flown in 1MC) flown by the EP. 
Note that this comparison only considered the display 
types with the 26 deg x 21 deg FOV. 

Mauchly’s test indicated that the assumption of 
sphericity had been violated for the main factor display 
type, % 2 (5)=37.370. Therefore, degrees of freedom 
were corrected using Greenhouse -Geisser estimates of 
sphericity (s-.73). The post-test paired-comparison 
SA-SWORD data indicated that display type 
(F(2. 19,74.5 1)=67.587, p<0.01) was highly significant 
for the pilot ratings of situation awareness. Contrasts 
showed four unique subsets for the SA ratings 1) 
uncollimated XVS with peripheral cues (highest SA), 
2) uncollimated XVS without peripheral cues, 3) 
collimated HUD with peripheral cues, and 4) 
collimated HUD without peripheral cues (lowest SA). 
The main factor motion and the second-order 
interaction between motion and display were not 
significant (p>0.05) for situation awareness. 

Mental Workload and Situation Awareness 
Discussion 

The subjective post-test ratings revealed that 
display color was a stronger influence on mental 
workload and SA than collimation. The workload and 
SA ratings also indicated that within the same display 
type (HUD or XVS) the presence of peripheral cues 
provided mental workload and SA improvements. 
These results were also supported by pilot comments 
during the post-test interviews. 

Concluding Remarks 

An experiment was conducted to investigate the 
use of XVS technologies as enabling technologies for 
future all-weather operations. The experimental 
objectives were to evaluate some of the design 


requirements of an XVS - without the need for 
forward-facing windows - and to determine the 
interaction of XVS and peripheral vision cues for 
terminal area and surface operations. . Objective 
results indicate that elimination of the visual segment 
of the approach had no adverse affects on localizer 
tracking as it was excellent regardless of the display 
type (XVS, HUD) and angle-of-attack (low, high) 
condition being evaluated or whether or not there were 
peripheral cues in the side windows. Motion cueing 
appears to have improved localizer tracking and 
touchdown sink rate across all displays. 

The data showed some evidence of a display 
collimation effect (i.e., when comparing XVS vs. 
HUD) for glideslope tracking performance. However, 
the magnitude of this effect did not impact landing 
performance. There also appears to be a peripheral 
cue/uncollimated display effect for glide slope tracking 
as improvements were found when peripheral cues 
were present while using the XVS display (HUD-like 
FOV and Full FOV). 

Touchdown performance was satisfactory with the 
HUD and XVS displays tested. All approaches 
resulted in landings that were within the current 
standard for landing touchdowns, but not 
unexpectedly, the longitudinal touchdown dispersions 
were too large to meet a 10 6 probability condition. 

The subjective data suggests that color, not 
collimation, was the primary effect for workload and 
situation awareness improvements found with the XVS 
display compared to the HUD display. 

In this experiment, it was impractical to 
completely divorce the collimation and display color 
variables, so the effects are confounded. However, if 
the collimation/color results found in this test could be 
validated, a less-complex non-collimated display could 
be suitable for an XVS -type system. Future research 
should include the validation of these results by flight 
testing in an operationally-realistic environment where 
vestibular and peripheral vision effects are accurately 
represented. 
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